When images of a plurality of input frames obtained by capturing the same subject have low resolution and there is positional deviation, the images are aligned and superimposed, thereby generating an image with high resolution. The process of generating a high-resolution image from the plurality of low-resolution images is called a super-resolution process.
Here, misalignment occurs in images in a case of imaging a stationary subject while moving a camera, or a case of imaging a moving subject with a fixed camera. In a case where the subject and the camera are completely stationary, no matter how many images are superimposed, the resolution is not increased.
For alignment (matching) of images, it is common to obtain an alignment parameter (geometric transformation matrix such as translation and rotation) using template matching or feature point matching. In addition, it is common to use a reconfiguration method for superimposing images. The reconfiguration method is a method of generating an estimated image of a high-resolution image, repeating update and evaluation of the estimated image so that the likelihood of the estimated image is maximized, and setting the converged solution of the estimated image as a final output. As the likelihood of the estimated image, there is [1] how much a low-resolution image obtained by degrading the resolution by shifting the position of the estimated image is close to an original input image, [2] whether the connection of pixel values in the estimated image is natural or not, or the like.
Meanwhile, in an automatic matching process in which images are automatically aligned, a process of finding and matching fiducial points is performed in alignment of images of a plurality of input frames, but mismatch due to erroneous detection of the fiducial point cannot be avoided. Especially, since subject blur is generated when a moving subject is captured at a slow shutter speed, the possibility of mismatch increases.
In order to reduce the possibility of mismatch due to the subject blur, the subject blur of the image may be removed before performing the super-resolution process. In order to remove the subject blur, a method of performing deconvolution is adopted, from a blur image and blur PSF, based on a model in which a result of convolving blur PSF on the original image without blur is a blur image (refer to PTL 1). This method makes it possible to obtain a blur removed image similar to the original image.